Retry policies

What is a retry policy?

A retry policy is a set of rules and guidelines that determine how to handle errors or failures that occur when an application or system attempts to perform a task or operation.

In general, a retry policy defines the number of times an operation should be retried, the interval between each retry, and the action to be taken if all retries fail. It is a strategy used to increase the chances of success for a particular operation by allowing the system to retry it and recover in the event of an error or failure.

For example, if a web application attempts to send a message to a server and fails due to a network error, the application may retry the operation a certain number of times with a specific time interval between each attempt. If the operation continues to fail after all the retries have been attempted, the retry policy may define a fallback action, such as notifying the user of the failure or logging the error for further analysis.

Retry policies are common in distributed systems, where network errors and service interruptions are common. They help to ensure that critical operations are performed successfully despite occasional errors or service disruptions.

Configuring a retry policy

gRPC retry options

• MaxAttempts : The maximum number of call attempts, including the original attempt. This value is limited by GrpcChannelOptions.MaxRetryAttempts which defaults to 5. A value is required and must be greater than 1.
• RetryableStatusCodes - A collection of status codes. A gRPC call that fails with a matching status will be automatically retried. For more information about status codes, see Status codes and their use in gRPC. At least one retryable status code is required.
• BackoffMultiplier : The backoff will be multiplied by this value after each retry attempt and will increase exponentially when the multiplier is greater than 1. A value is required and must be greater than zero.
• InitialBackoff : The initial backoff delay between retry attempts. A randomized delay between 0 and the current backoff determines when the next retry attempt is made. After each attempt, the current backoff is multiplied by BackoffMultiplier. A value is required and must be greater than zero.
• MaxBackoff : The maximum backoff places an upper limit on exponential backoff growth. A value is required and must be greater than zero.

Exponential Backoff

The initialBackoff, maxBackoff, and backoffMultiplier parameters determine the randomized delay before retry attempts.

The initial retry attempt will occur at random(0, initialBackoff). In general, the n-th attempt will occur at random(0, min(initialBackoff*backoffMultiplier**(n-1), maxBackoff)).

Let's try it

Retry policy in a channel

      private static GrpcChannel CreateChannel()
        {
            var methodConfig = new MethodConfig
            {
                Names = { MethodName.Default },
                RetryPolicy = new RetryPolicy
                {
                    MaxAttempts = 5,
                    InitialBackoff = TimeSpan.FromSeconds(0.5),
                    MaxBackoff = TimeSpan.FromSeconds(0.5),
                    BackoffMultiplier = 1,
                    RetryableStatusCodes = { StatusCode.Unavailable }
                }
            };

            return GrpcChannel.ForAddress("https://localhost:7226", new GrpcChannelOptions
            {
                ServiceConfig = new ServiceConfig { MethodConfigs = { methodConfig } }
            });
        }

Title

var client = new SpeakerServiceDefinition.SpeakerServiceDefinitionClient(channel);
 var client = client.GetById(new SpeakerFilterRequest() { Id = 5 });

Any clients created with the channel will apply the policy passted to the channel. Some retries are not valid, if the RPC becomes commited.

An RPC becomes committed in two scenarios:

• The client receives Response-Headers.
• The client’s outgoing message has overflowed the gRPC client library’s buffer.

Detect retries

The grpc-previous-rpc-attempts metadata can be used to determinine the retries. The intial call won't have it, but the retries will

• Is automatically added to retried calls and sent to the server.
• Value represents the number of preceding retry attempts.
• Value is always an integer.

Let's test this:

• Open a method service, and add this on the first method line:

Title

"Microsoft": "Warning"